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X.  Problea 

^ihls  report  1»  the  fourth  In  a  series  of  reports  summarlzlns  the 

literature  on  criteria. .  The  coirolete  plan  of  research  and  a  general 

\  ^ 

Introduction  to  the  problei^ls  uresentei  In  the  first  report,  and  In 

the  Program  Plan  for  PR  4904.^Erlefly,  an  attempt  will  be  made  to 

organize  and  summrrlze  publications  concerned  directly  with  the  problem 

of  selecting  and  constructing  criterion  meesure*.  Oiere  will  be  no 

effort  to  edit,  Interpret,  or  clarify  the  ideas^  As  in  Report  I,  the 

Problem  of  Definition,  the  iii  fer  tiTrTn  report  is  presented  in 

outline  form.  Intention  is  to  indicate  the  rarloas  rlewpolnts 

or  theories  expressed  and  provide  reference  to  the  author  (s)  advocating 

each  viewpoint.  Viewpoints  are,  therefore,  not  generally  detailed,  since 

it  is  expected  that  those  Interested  in  a  particular  aspect  of  the 

problem  will  first  be  concerned  with  general  coverage  of  the  area,. 


and  will  then  desire  to  consult  the  original  source  for  further  details, 
'whenever  a  viewpoint  has  been  taken  by  more  than  one  author,  all  are  listed 

II.  Procedure-A  description  of  the  method  used  In  gathering  the  material 
may  be  found  In  Report  ITumber  X  of  this  Aeries. 

III.  Summary  of  the  Literature 

This  section  will  list  the  various  measures  ^Ich  have  been  suggested 
in  the  literature  for  use  as  criteria.  will  be  broadly  classified 

according  to  type  of  measure.  That  Is,  the  specific  measure  will  be 
classified  into  the  following  five  broad  categories t  on-the-Job  records. 


y Study  of  the  Literature  on  Criteria:  X.  The  Problem  of  Definition. 


•tatus  or  achievement  of  a  particular  position,  score*  rcsult'xyr  from 
test  sltoatloQS,  evaluations  end  e.ssesBr.eats,  and  grades.  Crltlcless 
end  Ideas  concerning  these  specific  measures  will  be  presented  when  and 
as  these  crltlcls-us  appear  in  the  literature. 

A.  On-the-Job  record* 

1.  Data  on  aeasnres  relf.tlng  to  production: 

a.  Qp&ntlty,  e.g. ,  net  sale*,  nuasber  of  crj.ls  per  day,  units 
eonpleted,  etc.  (<,  9,  10,  11,  14,  34,  45,  £0,  68,  70, 

73,  84,  86,  90,  92,  94) 

l)  She  units  of  work  should  be  conpera'cle  for  all,  or  this 
factor  should  be  corrected  statietically,  (45)  Data  should 
be  collected  under  taalform  conditions  (84,  92,  103) 
over  an  extended  period  so  that  an  average  Is  obtained.  (10 ) 

b.  i^llty,  e.g. ,  errors,  breakage,  spoilage,  etc.  (9,  10,  11, 

45,  60,  68,  70,  86,  94) 

l)  The  suggestion  has  been  made  the  best  neasure  results 
where  all  errors  above  an  allowable  nlnlnum  'are 
autocatloally  detected  by  objective  inspection,  as  with  an 
electric  eye  cechanlsn.  (lOS) 

0.  Quantity  adjusted  for  quality  (11,  45,  90,  102)  e.g. , 

evaluate  tine,  natorial  or  cost  involved  In  the  imperfect  or 
spoiled  product.  (45) 

d.  Percent  efficiency,  i.e. ,  percent  ratio  of  earned  hours  to 

clock  hours  (earned  honre  moans  the  tlmo  set  by  eoaroany 

for  completing  a  unit  of  work,  on  the  basis  of  rational 

sinalysla,  experience  from  speed  of  work  on  similar  tasks, 

time  required  by  pace  setters,  tlae  snd  motion  Bsalyses*  and 

clock  hours  means  the  tlae  aotoally  required  by  the  worker.) 

(«,  90)  .3. 


2.  Accident  d£.ta.  (9,  11,  68,  70,  102) 

a.  Criticism:  Data  on  causes  are  ln«deq.aate;  accidents  are  often 
due  to  factors  beyond  the  control  of  the  Individual,  (66) 

3.  Absenteeism  (102) 

4.  Measures  related  to  training,  e.g. ,  time  required  to  train  employe#; 
(9,  10,  11,  45,  68,  86,  102,  103)  cost  of  training,  or  material 
used  during  training.  (9,  102) 

a.  Criticism: 

1)  It  is  difficult  to  equate  opportunity  to  learn.  (lOo) 

2)  Eelevsnce  to  ultimate  success  is  quest iozmahle  (86) 

5.  Measures  related  to  tenure. 

a.  detention  on  the  ^oh.  (4,  9,  10,  11,  45,  68,  70,  93,  111, 

102,  103) 

h.  ?reo.ucncy  of  job  changes.  (93,  95) 

0.  Promotions  (102)  end  rate  of  advancement.  (9,  10,  61,  68) 
l)  Criticism. 

(a)  ^ese  measures  are  dependent  on  business  conditions  (79) 

(b)  Seasons  for  leaving  are  vague,  contradictory, 
hard  to  obtain,  (87,  92) 

(c)  Tenure  data  is  only  available  for  post  left.  (93) 

6.  Composite  overfall  value  to  the  company: 

e.g. ,  "The  dollar  criterion"-  a  Job  analysis  is  accomplished 
to  reveal  the  variables  end  the  manner  and  extent  to  vhich  the  products 
end  behavior  affect  tl;e  efficiency  of  the  orginisation. 

Vhen  the  cost  accounting  evcduatlons  of  the  criterion  element's 
and  the  best  estimations  as  to  individual  differoxoes  are  available,  the 
iiq^ortanoe  of  each  criterion  element  can  then  be  estimated  as  the  product 


of  theao  two  fieares.  v 

At  th«  vel£;ht  b/  vihicb  the  munber  of  uaits  produced  Is  eultlplied 
this  equals  the  value  of  the  production  for  that  vork  unit.  The  value  of 
ell  the  vork  units  involved  la  the  Job  can  then  be  added  directly  to 
obtain  an  overall  composite. 

7.  Measures  related  to  Incooe 

a. 

1)  Salary.  (11,  40,  61,  68,  101,  103) 

a)  Average  for  a  specified  period  (69) 

b)  Salary  corrected  for  a^e  (85) 

2)  Comoissicns  and  bonuses.  (9,  10,  11,  68,  70,  73) 

3)  Oross  Samiass.  (25,  73) 

4)  Incone  after  a  certain  number  of  years.  (109) 

b.  Criticism:  Vage  increases  are  rately  given  on  the 
basis  of  merit  (103);  and  are  dependent  on  the  type  of  vork,  and- Union 
regulations.  (93) 

B.  Status  or  Achievement  of  a  Particular  Position. 

1.  Grade,  rank.  Job  level  (101 )  e.g. ,  the  Civil  Service  Grade 
or  Armed  Service  rank.  (85) 

3.  Achievement  of  a  certain  leadership  position,  e.g.,  captain 
of  a  team  (61)  executive  position.  (85) 

3.  Membership  in  professional  societies.  (9,  10,  11,  68) 

4.  Trade  status,  (joumeyman  vs.  apprentice)  (9,  10,  11,  68) 

6.  Alive-or-dead  -  Military  Criterion.  (86) 

6.  Besnlts  of  administrative  action,  e.g. ,  gradnation  or  non«  , 
graduation  or  ellnlnatlons  from  school.  (lOl) 


C.  Scores  reeultlo^  from  Test  Sltoetions.  (95) 

1.  Types  of  Test  Situations. 

a.  Tests  of  knowledge  and  Infomatloni  s.g. t  written  or 
oral  Standardized  achievement  tests.  (9,  10,  11,  68, 

70,  95,  101,  lOS) 

1}  Criticism: 

a)  ‘jQie  teste  usually  cover  only  a  part  of  the  Job. 
(28) 

b)  The  relationship  of  the  test  to  performance 
has  been  qusstionned  (lOl) 

o)  The  verbal  element  on  the  teat  may  be  a  biasing 
factor  (19). 

b.  Performance  Tests,  Job  or  vorh  sample  (ll,  45,  50,  75, 
90,  110,  119,  lao,  121)  e.s.,  the  standard  flight  In 
Air  Torce  (110,  119,  120,  121) 

1)  The  use  of  devices  for  measuring  and  recording 
results  has  ueen  sug;ested  e.g. ,  photographic, 
graphic,  sound  recorders  (50,  101,  119,  120,  121) 
e.g.  Interaction  chronograph  can  record  time  element 
In  a  situation  (24,  85).  Crltlelsmt  These  devices 
have  the  limitations  of  being  costly  and  time 
consuming  (120). 

0.  Ihraluatlons  and  Assessments. 

1.  Faotore  that  are  evaluated  or  assessed. 

a.  Product.  (25,  94)  ' 

b.  Perfoxnaaoe.  (24) 

e.  Degree  of  responsibility.  (9,  10,  11,  68,  69,  lOl) 
d.  Social  interaction. 


.6. 


^*r'. 


1)  Influence  on  fellow  worker*  (103) 

2)  Attitudes.  (28) 

e.  Henctlons  to  discipline  and  re^iulationa.  (28) 

f.  Job  Satisfaction.  (28,  40,  50,  68,  82,  86,  91,  93, 

101,  106,  109) 

e.  Personal  happiness.  (40,  109) 

2.  Individuals  who  evaluate. 

a.  Superiors.  (23,  45,  61,  79,  85,  92,  103) 

h.  Associates  (ecuals,  buddies).  (l25) 

1)  Advantages:  Usually  many  are  available,  .issociates 
are  better  able  to  assess  aspects  such  as  leader^ 
ship,  personality,  and  general  social  interaction. 
(125) 

c.  Subordinates.  (23) 

d.  Self,  (usually  evaluations  of  such  things  as  Job 
satisfaction,  adjustment  etc.)  (28,  40,  50,  68, 

82,  86,  91,  103,  106,  103) 

e.  Others:  Pamlly,  friends.  (28) 

f.  Combinations  of  the  above.  \ 

1)  Use  of  ratings  by  all,  each  of  whom  view  the 
Inslvldual  in  a  different  light.  (75) 

2)  Use  of  self  ratings,  and  evaluations  by  others  to 
cheek  each  other.  (122) 


3,  trechnioues  of  evaluation  and  assessment. 

«.  Batlngs  (graphics,  man-to-aan,  descriptive,  check  Ifst, 


1)  Mvaatiises  attrl'bated  to  ratlaga.  (63,  75) 

.a)  They  are  analytic 

h)  They  can  he  cmllclt  and  unaahlguous  and  can 
be  recoHed,  studied  and  Interpreted. 

c) -  They  are  uniform  and  standardized,  comparable, 

and  siuamari cable. 

d)  They  are  quantitative,  therefore  amenable  to 
< 

statistical  analysis,  and  correction  for  biases, 
etc. 

e)  They  can  be  el^en  periodically  and  systematically. 

f)  They  are  easy  to  use.* 

e)  They  represent  the  best  method  of  objectifying 
human  judgments,  particularly  of  less  tangible, 
but  important  variables. 

h)  They  enable  one  to  obtain  an  appraisal  of  the 
total  performance  in  all  its  essential  elements. 

2)  Recommendations  for  improvements  of  rating. 

a)  Use  of  factor  analysis  of  the  rating  fora  hcs 
been  suggested.  (13,  14} 

b)  Use  of  "critical  incident  technique*  to  get 

job  requirements  has  been  euggeated.  Description: 
Critical  requirements  of  a  job  or  activity 
are  established  through  direct  observations 
by  participants  in  or  supervisors  of  the 
activity.  A  critical  requirement  Is  crucial 

f 

in  that  It  has  been  responsible  for  outstandingly 
effective  or  definitely  unsatisfactory  performance 
of  an  important  part  of  the  job.  Critical 


rsqulreuents  fi.re  collected,  'ihese  are  eorted  into 
groups  of  eliallt-r  behavior  (content  analysis)  froa 
which  the  rating  scale  is  made.  (38,  39,  127,  129). 
Criticisa:  they  suffer  froa  the  usual  problecs  of 
rating  scales,  i-lso  no  validity  studies  are  available. 
(129) 

e)  Ihe  scale  should  be  adopted  for  the  specific  purpose 
and  situation.  (62) 

d)  She  scale  should  be  adopted  for  use  by  non  experts.  (63) 

e)  She  addition  of  new  adjectives  to  the  scale,  or  core 
points  may  improve  them.  (29) 

f)  She  mixing  up  of  scales  -  forcing  the  rater  to  study 
each  one  has  been  suggested.  (39) 

g)  ?reouency  charts,  shoving  how  many  ratings  of  each 
type  are  expected  should  be  used.  Scores,  should  be 
corrected  by  converting  them  to  standard  scores  having 
the  same  distribution  of  frequencies.  (S9) 

h)  She  use  of  more  raters  has  been  suggested.  (39) 

Z)  She  use  of  a  forced  choice  technique  has  been  suggested. 

(39) 

J)  She  classification  and  recording  of  observations  should 
improve  ratings.  (S9) 

k)  She  use  of  the  unweighted  average  of  impartial  Judges 
has  been  recommended.  (11) 
w.  Bankings  I  Ordeiwof-rnsrit.  (6,  55,  95,  97,  111) 
l)  A  suggested  method i  (12)  Separate  total  group  into  stth> 

groups.  Bank  vith^lfhe  sub-group.  Korgo  sub-group  by 


,i 


_ 


Modified  pi.ired  cor.parlson. 

2)  attributed  to  raaklngg ; 

a)  2he  method  has  face  validity.  (6) 

h)  aie  netl'^od  ia  reasonably  stable.  (6) 

c)  Puuiklngs  are  reasonably  unconteolaated  with 
regard  to  the  selection  variables.  (6) 

d)  The  method  is  easily  understood.  (12) 

e)  Puiters  like  end  have  confidence  in  the 
procedures.  (12) 

f)  The  suggested  method  above  can  be  epplled  to 
large  groups.  (12) 

g)  The  method  ref^uired  little  of  the  raters 
time,  (13) 

Konimations!  Sociometric  technltiues.  (38,  112) 

1)  Suggested  methods. 

e.)  nominations  of  people  with  whom  the  individual 
vould  like  to  work  (58,  85)  e.g. ,  fly  wing 
on  in  combat  (85,  113)  and  rea-sons  for 
nominations.  (112) 

b)  nominations  of  best  and  worst  workers.  (61,  85, 
86,  125) 

c)  Statement  of  whether  Individual  likes,  dislikes 
or  is  indifferent  to  every  other  worker.  (58) 

d)  iloEiination  of  an  individual  for  a  leadership 
position.  (61) 

2)  Advantages  attributed  to  nominations 

a)  nominations  are  mors  reliable  than  graphic 
ratings.  (136) 


9. 


b)  IThey  f.re  Eore  predictable  than  eoadenlq  grade. 

(136) 

c)  !IliQy  are  eapeclfJljr  useful  la  identifying  leaders. 
(58,  112) 

d)  !Il:ey  are  useful  as  iieasures  of  attitude  of 
vorkers  towards  each  other,  and  towards  canagesent. 

4.  Critlcisas  of  tbest  technlo.uea 

a)  Llfflcultlcs  exist  in  using  rating  scales  e.g. 
xmvllllngueas  of  rater#  to  rote.  (75,  101) 

b)  Hiey  lack  between  observer  reliability.  (S9, 

70,  116) 

c)  They  lack  validity  (39),  due  to  distortions, 
prejudices,  biases,  and  ai sunders tandlngs  as 
what  is  being  judged.  (28,  75,  93) 

d)  Sifflcultles  exist  in  the  interpretation  of 
ratings  due  to  difference#  la  standards  between 

\ 

raters,  (39,  92)  Isck  of  stead? rds  of  comparison 
between  jobs  (93),  failure  to  obtain  spread 
of  scores.  (39) 

e)  There  is  an  absence  of  definable  units  to  form 
a  continuous  scale  of  Judgment.  (28) 

f)  Plfflculties  In  weighting, exist.  (28) 

E.  Grades.  (60,  74,  85,  90,  95,  107,  109,  113) 

Grades  are  usually  a  composite  of  many  things;  mainly  a 
combination  of  evaluations  and  teat  scores.  "A  hodge-podge  of  many 

f 

eharaoteristlee  of  tho  individual,  the  Instructor,  the  course,  and  the 
eltnatloa,"  (107) 

$ 

1.  Types;  Specif io  course  grades  or  siunmary  grades  in  publio 
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schools,  traljilnc  schools,  corooratlon  schools. 

2.  Crlticlsia: 

&,  (They  suffer  f'Tosi  the  Ificlc  of  objectivity  slttilar  to 
ovtiluatlous  aaA  essessraeats.  (55  ,  75,  107) 
h,  P.elev.oice  to  ultlnate  Job  success  hts  been  ouestloned 
(4-1,  60,  93,  96,  101,  115) 
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